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The NuMI Off- Axis v e Appearance (NOi/A) project is a long-baseline neutrino experiment. It utilizes the 
NuMI neutrino beam at Fermilab and consists of two functionally-identical liquid scintillator filled detectors. 
The detectors are placed 14 milliradians off-axis from the beam and 810 km apart. A 209 ton prototype detector, 
the Near Detector On the Surface (NDOS), was built and began taking initial neutrino data in December 2010. 
NDOS is 110 milliradians off-axis from the NuMI beam and also records neutrinos from the Booster Neutrino 
Beam. As NDOS is in the commissioning phase, metrics are being developed to improve understanding of the 
detector as well as monitor the quality of data. Performance of the prototype detector will be presented. 



1. The NOi/A Experiment 

The NuMI 1 Off- axis v e Appearance (NOz^A) experiment will be able to address many questions about the 
parameters of neutrino oscillation. NOz^A will be an order of magnitude more sensitive to values of sin 2 (2^13) 
than current limits near 0.1, as shown in Figure [I] It will be able to probe the mass hierarchy as well as examine 
if oscillations violate Charge- Parity. NOz/A will also make precision measurements of #23 and lAra^l- 




Figure 1: NOz^A 3a sensitivity as a function of 613. The blue curves corresponds to normal mass hierarchy, while the 
red curves correspond to the inverted mass hierarchy. These sensitivities are based on 3 years of running each with a 
neutrino and ant i- neutrino beam. The dotted curves represent higher beam intensities but the solid curves are with the 
planned 700 kW beam intensity. 

NOz^A consists of two detectors, one located at Fermilab in Batavia, IL, (called the near detector) and one 
located at Ash River, MN near the Canadian border and Voyageurs National Park (called the far detector). 
These detectors are functionally-identical except for overall size. The near detector will be about 200 tons, 
while the far detector will be about 14 kton. These detectors are placed to intercept the neutrinos from the 
NuMI beam created at Fermilab [2 . 

The NOz^A detectors have a cellular structure. Each cell is a reflective PVC tube that is filled with liquid 
scintillator (mineral oil doped with about 5% pseudocumene). Within the cell is a loop of wave-length shifting 



1 Neutrinos at the Main Injector [I] 
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fiber whose ends are connected to an Avalanche Photodiode (APD). For the far detector, the dimensions of the 
cell are 4 cm x 6 cm x 15 m. The near detector cells have the same cross-sectional area, but are 4 m long. These 
dimensions result in 0.15 radiation lengths per plane of cells. The planes of cells are orientated perpendicular to 
the direction the neutrinos are traveling and each plane alternates from being horizontally or vertically aligned 
in the perpendicular plane. This allows for full three-dimensional reconstruction. 

When a neutrino interacts within the detector volume, charged particles are created. As charged particles 
travel through the liquid scintillator, they create photons. These photons are absorbed by the wave-length 
shifting fiber and are transported to the fiber end, where they are read-out by the APD. The APD amplifies 
the signal by a factor of one hundred, a level capable of being read by sensitive, low-noise electronics. 

Each APD has 32 pixels which allows it to read out one module. Each APD is then connected to a Front 
End Board (FEB). The FEB digitizes the signal from the APD and sends it to a Data Concentrator Module 
(DCM). Each DCM is capable of reading out up to 64 FEBs. This signal is then passed to the rest of the data 
acquisition system [2]. 



2. The Near Detector On the Surface (NDOS) 

The NDOS is the prototype detector for the NO^A detectors. At 209 tons, it is approximately the size 
of the future near detector and is functionally-identical. It is located on the surface at Fermilab instead of 
underground, as the near detector will be. The NDOS is currently taking neutrino data. Having a prototype of 
this scale has proved invaluable in preparation for construction of the real detectors. It has allowed us to test 
and improve everything from assembly procedures to the timing systems. 

NDOS detects neutrinos from the NuMI beam, currently running for the MINOS experiment [3 , as well as 
from the Booster Neutrino beam. The Booster Neutrino beam is primarily for the MiniBooNE experiment [4]. 
The NDOS detector is 110 mrad off- axis from the NuMI beam and the detector axis is in the same plane as the 
beam axis. The NDOS is nearly on-axis relative to the Booster beam and the detector axis is rotated relative 
to the beam axis by 23°. 



3. Monitoring Systems 

The NDOS has five levels of data monitoring: the Data Acquisition (DAQ) Monitor, the Memory Viewer, 
the Event Display, the Online Monitor, and DataCheck. The first four are real-time monitors, while DataCheck 
has a delay of about one hour. Both Online Monitor and DataCheck report run metrics. Under stable running 
conditions, runs are about a day in length, with subruns lasting approximately one hour. Subruns consist of 
trigger windows. Everytime the system triggers on either a neutrino spill or a regular, 10 Hz cosmic trigger, a 
500 /isec time window of data is written out. The near and far detectors will have much shorter trigger windows; 
the longer windows for NDOS allowed us to increase the load on the DAQ system. 

3.1. DAQ Monitor 

The first level of monitoring is done by the DAQ Monitor. It monitors the health and performance of the 
entire DAQ System, from the DCMs to the network connections. It uses Ganglia as its base. Ganglia is a 
third-party, easily customizable, open-source software that specializes in tracking statistics like memory and 
network usage. NOz/A has created metrics such as data and trigger rates, data sizes, error states, and the 
level of data corruption. All of these metrics are written to a database and accessed through a web interface 
which allows them to be plotted interactively. The example display shown in Figure [2] is used to monitor the 
status and performance of a DCM. In general, custom monitored metrics can be used to troubleshoot problems, 
optimize performance, and diagnose trends over both short and long time periods. 

3.2. Memory Viewer 

Memory Viewer is the next level of monitoring. Memory Viewer displays colored bytes of the raw data 
from the detector, giving an intuitive sense of the data flow. Using pattern matching, one can tell if the run 
is operating normally. Memory Viewer, as well as many of the other monitoring systems, relies on Event 
Dispatcher. The Event Dispatcher is a server program running on a DAQ host which streams event data to 
clients, which connect to it via a TCP socket. Memory Viewer connects to this stream and displays the bytes. 
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Figure 2: An example of plots generated by the DAQ Monitor to monitor a DCM. For all the plots, the horizontal axis 
is current time in hours. Data Dev Buffers Used and Max Data Dev Buffers Used plot the percentage of circular buffer 
capacity for the DCM device driver that was occupied on average and at the peak for each time interval. Monitoring 
these values ensures we are not using all the available capacity, which could result in lost data. MicroSlice Rate is the 
rate in Hz of microslices for each time interval. Microslices are concatenated amounts of data and the rate should be 
steady, since it is a configured parameter. The MicroSlice Size and MicroSlice StDev report the average size in bytes 
and standard deviation of the size of microslices. The size is a reflection on the amount of data we are collecting; the 
standard deviation is a measure of how steady this data rate is. Millislices are larger concatenations of data. We monitor 
the MilliSlice Rate and MilliSlice Size in a manner analogous to microslices. Lastly, we monitor the Missing MicroSlice 
Rate in Hz. This should be zero if we are not losing data. 




Figure 3: This is the graphical user interface of the Memory Viewer. The raw data is displayed as colored bytes, giving 
an intuitive sense of the data. Using pattern matching, one can quickly assess the run status. 



3.3. Event Display 

Event Display is a detector view of the trigger windows from Event Dispatcher. It shows both of the two- 
dimensional views: the view from the top of the detector and the view from the side. Each channel is represented 
in one of the views as a grey rectangle. If the channel had a hit during the trigger window, it is colored. The 
color can indicate the relative time of the hit, as in Figure [4j or the relative charge of the hit. The beam enters 
the front of the detector, the left-hand side of the Event Display. Towards the back of the detector, we have 
regions that are either uninstrumented or only partially instrumented. Finally, the muon catcher is represented 
at the right side of the Event Display; its planes are interspersed with steel so the instrumentation is less dense. 
Figure [4] shows cosmics rays entering the detector. 



3.4. Online Monitor 



Online Monitor displays run metrics such as the number of active channels and channel occupancies in real 
time. Figure [5] shows Online Monitor displaying the occupancies of the FEBs which gives an indication of which 
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Figure 4: The Event Display is a detector view of the data. It shows both the top and side views of the detector. Each 
channel is a grey rectangle. If the channel had a hit, it is colored. In this case, the color indicates the relative time of 
the hits. The beam enters the front of the detector on the left-hand side of the Event Display. The detector has regions 
that are uninstrumented or only partially instrumented. The muon catcher is located at the back of the detector and 
has layers interspersed with steel planes. 




Figure 5: Plot of FEB occupancies from Online Monitor. Each rectangle represents one FEB and its color is the relative 
numbers of hits as given by the scale at the right of the figure. This gives an indication of each FEB's noise levels. 



FEBs are noisier than others. Online Monitor is broken into two processes to increase system stability. Online 
Monitor Producer reads in information from the Event Dispatcher and processes it to create run metrics. Online 
Monitor Viewer is a graphical user interface that communicates with the Producer through a ROOT TMapFile. 
The Viewer is responsible for actually displaying the information. 



3.5. DataCheck 



DataCheck is the final level of monitoring. It is an offline software tool and has a delay of about one hour 
from the end of a subrun. DataCheck has the ability to look at metrics subrun by subrun, or cumulatively 



Proceedings of the DPF-2011 Conference, Providence, RI, August 8-13, 2011 



5 




Figure 6: Number of active channels in NDOS. In May 2011, we began actively removing questionable channels for 
further study. Intermittent drops are due to running with a partial detector during commissioning. A fully-instrumented 
NDOS would have 15,900 channels. 



across multiple subruns or runs to find trends in the data. It also has a web interface that uses PHP to make 
plots from a database. DataCheck monitors metrics, such as the number of active channels, the average number 
of hits per channel, the number of active FEBs, and the relative times of neutrino candidates. Section [4] reports 
some of the findings from DataCheck. 



4. Performance of the NDOS 

One fundamental metric of the performance of the NDOS is the number of active channels. Active channels 
are defined as those that are instrumented and operating within acceptable parameters. From Figure [6j one 
can see that channels were steadily installed throughout the winter of 2010 and spring of 2011. However, in 
May 2011, we began actively removing all channels that displayed questionable behavior. This has allowed us 
to fully understand all the problems with our channels before we begin procurement for the far detector. One 
may also note the drops in channel level throughout the run period. This is due to frequent running with only 
parts of the detector while work is being done on other sections. For reference, a fully-instrumented NDOS 
would have 15,900 channels. 

Another basic metric is the number of hits per trigger window per active channel. One can see from Figure 
[7] that it is generally 5%. When initally commissioning NDOS, we set high thresholds and the average number 
of hits was lower. Once we understood our system, we lowered our thresholds to more reasonable levels, which 
accounts for the rise in average number of hits. One can see that sometimes the NDOS gets very noisy. This 
is due to humidity effects, electronics noise, etc. These effects are under study and we are working to mitigate 
them in the near and far detectors. 

This noisy behavior is also reflected in the rate of FEB shutoffs. Our system is designed to shutoff a FEB 
during a run if the data buffer overflows. The FEB then remains off for the rest of the run and is re-initialized 
when the next run begins. Data buffers usually overflow due to noisy APDs. Thus, high rates of FEB shutoffs 
are a concern. Initially, we noticed noisy run conditions seemed to be correlated with high humidity in the 
detector hall. Then we began to cool our APDs to — 15° C to reduce the noise levels. However, when we did 
this on the NDOS, some channels became unusable. It was found that many had water on the surface of the 
APD due to condensation from air infiltration. We are investigating and redesigning our electronics installation 
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Figure 7: Hits per trigger window (500 /isec) per active channel. Initial running had high thresholds and thus low values 
of this metric. After lowering the thresholds, we have been relatively stable at 5%. Jumps are due to noisy periods in 
the NDOS that are under investigation. 



procedure and circuit boards to ensure that the APDs are in a sealed environment. 

Using rough activity, fiducial, and direction cuts, DataCheck can quickly find neutrino candidates. The time 
distribution of these candidates relative to the trigger window is plotted in Figure [8] A peak in this distribution 
within the 10 /isec NuMI beam spill indicates the timing and triggering systems are operating as expected. 
Figure [8] contains one week's worth of running and a peak is clearly visible. This allows us to verify if our 
systems are working on a reasonably short time scale. We also track the Booster neutrino peak; however, with 
the lower statistics, it takes more than a week to see a peak. 

5. Summary 

In summary, the NDOS has been taking data for 9 months. The NO^A collaboration will begin building the 
near and far detectors this winter. We have viable systems in place to monitor both detector and data quality. 
These systems continue to aid us in commissioning NDOS and preparing for future detector construction. 
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Figure 8: Times of NuMI neutrino candidates within the trigger window (500 //sec) for one week's running. Neutrino 
candidates are found using rough activity, fiducial, and direction cuts. If a peak is seen in the distribution within the 10 
//sec NuMI beam spill, we have verified that the trigger and timing systems are working as expected. 



